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Abstract  This  paper  discusses  efforts  at  the  Idaho  National  Engineering  and 
Environmental  Laboratory  (INEEL)  to  develop  a  control  architecture  for 
mixed  teams  of  ground  and  air  vehicles.  Our  approach  has  been  to  consider  the 
air  vehicles,  ground  robots  and  humans  as  team  members  with  different  levels 
of  authority,  different  communication,  processing,  power  and  mobility 
capabilities  and  also  different  perceptual  capabilities.  The  paper  examines  the 
current  control-architecture  and  interface  with  special  attention  to  the  role  of  a 
collaborative  workspace  in  enabling  mixed-initiative  interaction  between 
humans  and  heterogeneous  teams  of  robotic  vehicles. 
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1.  INTRODUCTION 

The  potential  exists  to  extend  the  capabilities  of  autonomous  operations 
through  synergistic  interaction  between  ground  and  air  vehicles. 
Autonomous,  unmanned  ground  and  air  vehicles  may  complement  one 
another  in  powerful  ways.  For  instance,  ground  robots  can  encounter  a  rich 
slice  of  their  local  environment,  yet  are  often  limited  by  the  lack  of  an 
accurate  ‘god’s  eye’  view.  To  address  this  limitation,  air  vehicles  can  be 
used  to  travel  large  distances,  gathering  intelligence  and  reconnaissance  data, 
which  helps  to  orient  ground  robots  within  the  global  environment.  In  turn, 
ground  robots  can  support  air  vehicles  by  carrying  heavy,  power  hungry 
sensors,  sharing  computational  processing,  relaying  communications, 
providing  close  up  visual  and  perceptual  reconnaissance  and  even  through 
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close  proximity  interactions  such  as  deploying,  refueling  and  retrieving  air 
vehicles.  This  paper  discusses  recent  efforts  at  the  INEEL  to  develop  a 
control  architecture  that  supports  mixed-initiative  human  -  robot  teaming  for 
a  small  team  of  robots.  The  prototype  control  architecture  has  been 
implemented  on  a  number  of  ground  vehicles  and  is  currently  being 
extended  to  a  number  of  real  and  simulated  assets  through  collaboration  with 
the  US  ARMY  Simulation  and  Training  Command  and  the  Tactical 
Technology  Office  of  the  Defense  Advanced  Research  Projects  Agency. 

This  paper  describes  efforts  to  create  a  dynamic  umbrella  of  adjustable 
autonomy  control  that  can  support  a  spectrum  of  team  interactions 
throughout  a  mission  as  individual  capabilities  change  and  needs  arise.  As 
workload,  data  link  connectivity,  knowledge  and  action  potential  change, 
dynamic  autonomy  enables  each  member  to  adjust  its  level  of  autonomy  on 
the  fly,  leaning  on  its  own,  intrinsic  intelligence  as  needed.  The  vision  is  that 
of  a  continuously  adapting  control  architecture  where  all  team  members, 
including  humans,  air  vehicles  and  ground  vehicles,  are  empowered  to  self¬ 
monitor,  maintain  awareness  of  the  environment  and  task,  and  actively  and 
cooperatively  pursue  mission  objectives  using  a  common  substrate  -  a 
collaborative  workspace.  By  sharing  information  about  the  task  and 
environment  at  an  appropriate  level  of  abstraction  and  through  a  common 
representation,  each  team  member  is  enabled  to  support  one  another  with 
information,  tactical  mission  support  and  even  more  tangible  forms  of 
assistance  such  as  offering  physical  transportation  or  bringing  new  supplies. 


2.  RESEARCH  PROBLEM 

The  nature  of  21st  Century  asymmetric  warfare  requires  systems  that 
operate  effectively  in  highly  decentralized  environments,  making  maximum 
use  of  local  information  and  resources  to  provide  overwhelming  global 
results.  However,  the  benefits  of  mixed  teams  can  easily  be  overshadowed 
by  communication,  processing  and  control  challenges  inherent  to  developing 
and  deploying  mixed  teams  of  humans,  ground  robots  and  air  vehicles.  For 
mixed  teams  of  humans  and  heterogeneous  robots  to  be  realized  there  is  a 
considerable  need  for  further  research  into  the  problems  of  distributed 
communication,  collaborative  perception,  and  shared  control. 

For  the  human,  mixed  teams  present  particularly  difficult  HRI  questions: 

•  Do  humans  lead  all  aspects  of  a  task?  Can  a  machine  give  orders? 

•  What  needs  to  be  communicated  and  at  what  level  of  abstraction? 

•  Can  initiative  shift  throughout  the  task? 

•  Can  team  roles  shift  to  reflect  changing  capabilities? 

•  When  can  machines  say  no  to  humans  or  to  other  machines? 
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The  operator  must  not  only  understand  the  robotics  force  in  terms  of 
where  they  are,  but  s/he  must  understand  what  they  are  doing  now  and  what 
they  will  likely  do  in  the  near  future.  Situation  awareness  is  a  critical 
element  in  decision  making,  especially  in  highly  dynamic  situations  that  are 
outside  of  normal  operations.  A  loss  of  situation  awareness  will  likely  result 
in  slower  detection  and  reaction  times  as  the  operator  struggles  to  re-orient 
him  /herself  with  the  current  situation.  Consequently,  our  work  has  focused  a 
great  deal  of  effort  on  developing  appropriate  interfaces  and  possibilities  for 
human  monitoring  and  intervention.  Mixed  teams  will  require  human 
interfaces  that  merge  traditional  military  map-based  planning  with  graphical, 
iconographic  capabilities  which  permit  the  operator  to  interact  with 
abstracted  representations  of  the  robots’  experience  in  order  to  monitor  the 
vehicles,  maintain  situation  awareness  and  task  the  autonomous  assets. 

Most  importantly,  the  individual  vehicles  must  be  imbued  with  the 
intrinsic  intelligence  necessary  to  exploit  on-board  sensing  and  serve  it  up  to 
be  fused  with  data  from  other  vehicles.  Establishing  a  common 
communication  substrate  will  be  a  critical  issue  for  teams  of  diverse 
vehicles.  Especially  for  heterogeneous  air  and  ground  vehicles,  this  objective 
requires  a  means  to  fuse  a  wide  variety  of  sensing  from  disparate  modalities 
and  perspectives.  Current  approaches  that  assume  high-bandwidth, 
continuous  data  connectivity  between  air  and  ground  vehicles  are  wholly 
inappropriate  for  a  battlefield  where  communications  will  inevitably  be  lost, 
jammed  or  intercepted. 


3.  MIXED-INITIATIVE  TEAMS 

Teleoperated  systems  have  often  failed  to  address  the  limitations  of 
telepresence  inherent  to  current  communication  technologies.  On  the  other 
hand,  attempts  to  build  and  use  autonomous  systems  have  failed  to 
acknowledge  the  inevitable  boundaries  to  what  the  robot  can  perceive, 
understand,  and  decide  apart  from  human  input.  Both  approaches  have  failed 
to  build  upon  the  strengths  of  the  robot  and  the  human  working  as  a  cohesive 
unit.  Alternatively,  mixed-initiative  systems  can  support  a  spectrum  of 
control  levels.  Mixed-Initiative  robots  should: 

•  Possess  intrinsic  intelligence,  knowledge  and  agency. 

•  Have  the  ability  to  protect  humans,  environment  and  self. 

•  Dynamically  shift  levels  of  initiative  to  accept  different  levels  and 
frequencies  of  intervention. 

•  Recognize  when  help  is  needed  (from  human  or  machine) 

•  Learn  from  these  interactions 
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This  paper  discusses 
recent  efforts  at  the 
INEEL  to  develop  a 
control  architecture  that 
supports  mixed-initiative 
human  -  robot  teaming 
for  a  team  of  robots 
including  a  number  of 
all-terrain  ground  robots. 
At  the  current  time,  two 
ATRVJr’s  (see  figure  1) 
and  two  ATRVmini’s  are 
being  used.  Work  is 
underway  with  the  U.S. 
Army  Simulation  and 
Training  Command  to  include  simulated  vehicles  and  with  DARPA  TTO  to 
interface  with  real  world  Micro  Air  Vehicles  developed  under  the  MAV 
program. 


Figure  1.  One  of  the  current  team  members  -  a 
modified  ATRVJR  platform  from  iRobot. 


3.1  Platform-Centric  Intelligence 

In  order  to  enable  dynamic  autonomy,  the  INEEL  has  developed  sensor- 
suites  and  fusion  algorithms  for  sensing,  interpreting,  responding  to  the 
environment.  The  robotic  systems  are  able  to  autonomously  perform 
mission-level  task  decomposition  based  on  commands  from  other  robots  or 
human  supervisors.  The  robots  are  then  able  to  carry  these  tasks  out  by 
invoking  robust,  reactive  behaviors  such  as  obstacle  avoidance,  pursuit, 
escape,  get-unstuck,  search  and  explore. 

To  accomplish  these  behaviors,  each  robot  must  fuse  a  variety  of  sensor 
information  including  inertial  sensors,  compass,  wheel  encoders,  laser  range 
finders,  computer  vision,  thermal  camera,  infrared  break  beams,  tilt  sensors, 
bump  sensors,  and  sonar.  The  robots  use  this  fused  sensor  information  to 
navigate  safely,  placing  minimal  limits  on  the  user  to  take  the  robot’s 
immediate  surroundings  into  account.  The  robots  also  continuously  assess 
the  validity  of  its  diverse  sensor  readings  and  communication  capabilities. 
The  robot  will  refuse  to  undertake  a  task  if  it  does  not  have  the  ability  (i.e., 
sufficient  power  or  perceptual  resources)  to  safely  accomplish  it. 
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Not  only  must  the  robot 
be  able  to  fuse  information 
for  itself  to  utilize,  it  must 
also  be  able  to  abstract 
these  percepts  and 
communicate  them  to 
human  supervisors  and 
other  robots.  Figure  2 
shows  how  the  current 
interface  communicates 
obstacles  (red  ovals)  and 
resistance  to  motion 
(pulses)  encountered  by  an 
individual  robot.  The  same 
information  which 

populates  this  interface 
module  is  also  received  by 
other  robots  in  the  team 
which  can  choose  to  use  or 
ignore  it  as  they  see  fit. 

Figure  2.  Feedback  on  an  individual  robot’s 
immediate  surroundings. 


3.2  Multi-Modal  Communications 

In  order  to  enable  robust  team  work  within  critical  environments  such  as 
Military  Operations  in  Urban  Terrain  (MOUT),  it  is  not  sufficient  to  assume 
that  high-bandwidth  connections  such  as  wireless  ethemet  exist  between  all 
team  members.  Rather,  the  dynamic  team  structure  must  be  supported  by  an 
equally  dynamic  communication  infrastructure.  Consequently,  the  INEEL 
has  been  working  to  develop  a  specialized  protocol  and  multi-modal 
communication  architecture  to  support  dynamic-autonomy.  Our  approach 
has  been  to  invest  each  vehicle  with  sufficient  agency  and  intelligence  so 
that  they  can  share  information  at  an  appropriate  level  of  abstraction.  Our 
robots  do  not  ever  need  to  share  raw  data,  but  instead,  communicate  using 
extensible,  parameterized  message  strings.  These  message  strings  are  used 
by  other  robots  in  the  team  and  are  simultaneously  used  to  populate  the 
human  interface.  Although  it  is  presented  differently  to  the  human 
teammates,  the  basic  form  of  representation  is  the  same  across  the  team.  The 
protocol  has  been  developed  to  permit  abstracted  percepts  and  commands  to 
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travel  over  a  low  bandwidth  communications  channel  of  no  more  that  9600 
baud.  This  allows  the  system  to  use  cell  phone  communications  and  non  line 
of  sight  radio  links  that  can  travel  for  up  to  60  miles.  Robust  teamwork  must 
be  built  on  robust  communications. 

3.3  Dynamic  Leadership 

Rather  than  conceive  of  machines  as  tools  or,  on  the  other  hand,  as  totally 
autonomous  entities  that  act  without  human  intervention,  it  is  more  effective 
to  consider  the  machine  as  part  of  a  dynamic  human-machine  team.  Within 
this  team,  each  member  is  invested  with  agency  -  the  ability  to  actively  and 
authoritatively  take  initiative  to  accomplish  task  objectives.  Each  member 
has  equal  responsibility  for  performance  of  the  task,  but  responsibility  and 
authority  for  particular  task  elements  shifts  to  the  most  appropriate  member, 
be  it  human  or  machine.  For  instance,  in  a  remote  situation,  the  robot  may  be 
in  a  much  better  position  than  the  human  to  react  to  the  local  environment, 
and  consequently,  the  robot  may  take  the  leadership  role  regarding 
navigation.  As  leader,  the  robot  can  then  “veto”  dangerous  human 
commands  to  avoid  running  into  obstacles  or  tipping  itself  over. 

Flowever,  the  ability  to  shift  a  robot  in  and  out  of  the  leadership  role 
presents  a  conundrum.  The  user  comes  to  rely  on  the  self-protective 
capabilities  of  the  robot  and  yet,  at  times,  must  override  them  to  accomplish 
a  critical  mission.  The  benefits  of  allowing  the  team  members  to  change 
roles  within  the  team  significantly  increases  team  flexibility  and  reliability  in 
task  performance.  Flowever,  if  the  interface  and  human-robot  system  are  not 
designed  in  accordance  with  critical  principles  of  human  factors  in  mind, 
dynamic  role  changing  may  result  in  mode  confusion,  loss  of  operator 
situation  awareness,  loss  of  operator  confidence  in  assuming  supervisory 
control,  and  degraded  and  potentially  catastrophic  performance  (Abbott  et 
al.,  1996).  Systematic  human-centered  design  is  necessary  to  insure  that  the 
robot  autonomy  conforms  to  the  ways  in  which  humans  assign  and  manage 
tasks. 


4.  LEVELS  OF  HUMAN  INTERVENTION 

Within  the  last  several  years,  researchers  have  begun  in  earnest  to 
examine  the  possibility  for  robots  to  support  multiple  levels  of  user 
intervention.  Much  of  this  work  has  focused  on  providing  the  robot  with  the 
ability  to  accept  high  level  verbal,  graphical,  and  gesture-based  commands 
(Perzanowski  et  al.,  2002;  Kortenkamp  et.  al.,  1996;  Voyles  and  Khosla 
1995).  Others  have  implemented  robots  that  understand  the  limitations  of 
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their  autonomous  capabilities  and  can  query  the  user  for  appropriate 
assistance  (Fong  et  al.,  2002;  Murphy  and  Rogers,  1996).  Goodrich  et  al. 
(2001)  have  performed  experiments  which  involve  comparing  the 
performance  of  human-robot  pairs  using  different  modes  of  human 
intervention.  However,  very  little  work  has  emphasized  true  peer  to  peer 
interactions  where  the  robot  is  actually  able  to  shift  modes  of  autonomy  as 
well  as  the  user.  Scholtz  (2002)  discusses  the  need  for  this  kind  of  peer-peer 
interaction,  and  provides  categories  of  human  intervention  including 
supervisory,  peer  to  peer  and  mechanical  interaction  (e.g.  teleoperator). 

Our  research  to  date  has  developed  a  control  architecture  that  spans  these 
categories,  supporting  the  following  modes  of  remote  intervention: 
Teleoperation,  Safe  Mode,  Shared  Control,  and  Full  Autonomy.  Within 
teleoperation  mode,  the  user  has  full,  continuous  control  of  the  robot  at  a  low 
level.  The  robot  takes  no  initiative  except  to  stop  once  it  recognizes  that 
communications  have  failed.  Within  safe  mode,  the  user  directs  the 
movements  of  the  robot,  but  the  robot  takes  initiative  to  protect  itself.  The 
robot  assesses  its  own  status  and  surrounding  environment  to  decide  whether 
commands  are  safe.  Within  shared  control  mode,  the  robot  takes  the 
initiative  to  choose  its  own  path,  responds  autonomously  to  the  environment, 
and  works  to  accomplish  local  objectives.  Within  the  full  autonomy  mode, 
the  robot  performs  global  path  planning  to  select  its  own  routes,  requiring  no 
user  input  except  high-level  tasking  such  as  "follow  that  target"  or  "search 
this  area.”  The  human  user  can  switch  between  modes  to  cope  with  different 
components  of  the  task. 

For  each  of  these  levels  of  autonomy,  perceptual  data  is  fused  into  a 
specialized  interface  (shown  in  Figure  3)  that  provides  the  user  with 
abstracted  graphical  and  textual  representations  of  the  environment  and  task 
that  are  appropriate  for  the  current  mode.  The  robot  relays  a  great  deal  of 
synthesized,  high-level  information  (including  suggestions  and  requests  for 
help)  to  the  user  in  a  textual  form  using  the  feedback  textbox  within  the 
image  window.  Also  note  that  the  robot  provides  textual  reports  on 
environmental  features  at  the  bottom  of  the  map  window  and  reports  on 
communications  status  at  the  bottom  of  the  robot  status  window.  The  robot 
status  window  provides  a  variety  of  information  about  the  status  of  an 
individual  robot  including  pitch  and  roll,  power,  heading,  speed  and  a  fusion 
of  this  information  into  a  single  measurement  of  “health.”  The  user  can  shift 
an  individual  robot  into  shared  or  teleoperation  mode  and  then  move  the 
robot  by  touching  the  arrows  by  using  a  joystick.  Also,  it  is  possible  to  pan 
and  tilt  the  camera  on  each  individual  robot  automatically  by  touching 
regions  of  the  visual  image. 
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Figure  3.  Current  Interface  for  Human  Intervention 


5.  COLLABORATIVE  WORKSPACE 

The  fundamental  aspect  of  a  human  team  that  distinguishes  it  from  a 
simple  group  is  the  presence  of  a  shared  goal.  We  must  have  some  mwans  to 
represent  this  common  goal  with  a  common  form  of  representation  that  is 
meaningful  to  all  members.  Effective  teams  typically  cooperate  and 
anticipate  the  needs  of  teammates  via  a  shared  mental  model  of  the  task  and 
current  situation  (Yen  et  al.,  2001).  If  we  want  humans,  air  vehicles  and 
ground  vehicles  to  work  as  a  team,  we  need  to  develop  an  appropriate  level 
of  discourse,  including  a  shared  vocabulary  and  a  shared  cognitive 
workspace,  collaboratively  constructed  and  updated  on  the  fly  through 
interaction  with  the  real  world. 

We  have  chosen  to  address  this  need  by  building  a  map  that  consists  of 
terrain  overlaid  with  semantic  abstractions  generated  through  autonomous  or 
user-assisted  recognition  of  environmental  features.  The  current  mapping 
algorithm  is  based  on  simultaneous  localization  and  mapping  (SLAM)  work 
done  at  the  Naval  Research  Laboratory.  (Yamauchi  et  al.,  1999)  At  the 
present  time,  we  have  successfully  implemented  the  algorithm  on  individual 
robots  and  integrated  it  with  the  control  architecture  and  communication 
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protocol.  Since  this  SLAM  algorithm  was  not  designed  to  localize  multiple 
robots  with  unknown  map  locations,  the  current  research  effort  assumes  that 
GPS  will  be  available,  which  will  allow  each  robot  to  add  information  to  a 
common  map  based  on  their  known  position.  Research  to  identify  and 
incorporate  other  algorithms  for  collaborative  mapping  is  ongoing. 

For  the  user,  the  current  map  provides  point-and  click  user  validation  and 
iconographic  insertion  of  map  entities.  The  user  can  verify  or  remove 
entities,  which  have  been  autonomously  added  and  can  add  new  entities, 
which  the  robots  were  unable  to  find.  To  test  this  capability,  we  outfitted  one 
of  the  ground  robots  with  a  metal  detector,  which  allowed  it  to  autonomously 
identify  the  presence  of  metal  objects  and  add  them  to  the  map  as  red  circles. 
In  addition,  the  human  can  add  a  number  of  other  entities  such  as  human 
victims  (shown  in  figure  4  as  the  purple  circles). 


Figure  4.  Current  Collaborative  Workspace 


This  real-time  semantic  map,  constructed  collaboratively  by  human  and 
machines,  serves  as  the  basis  for  a  spectrum  of  mutual  human-robot 
interactions  including  tasking,  situation  awareness,  human-assisted 
perception  and  collaborative  environmental  “understanding.”  In  addition,  the 
collaborative  cognitive  workspace  can  be  utilized  by  dynamic  planning 
systems  to  orchestrate  synergistic  interactions  between  air,  ground,  surface 
and  underwater  vehicles.  For  instance,  air  vehicles  can  provide  the 
workspace  with  a  global,  “god’s  eye”  view  of  terrain,  aiding  the  ground 
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vehicles  as  they  autonomously  plan  paths.  Conversely,  the  ground  vehicles 
can  provide  more  detailed,  precise  intelligence,  surveillance  and 
reconnaissance  (ISR)  of  areas  of  interest  (AOI). 

The  collaborative  workspace  is  also  a  palette  on  which  to  interleave 
multiple  levels  of  human  intervention  into  the  functioning  of  a  multi-robotic 
system.  The  human  can  assist  in  identifying  and  classifying  targets  in  the 
environment  using  an  interactive  visual  display.  Collaborative  construction 
of  the  map  enhances  each  individual  team  member’s  understanding  of  the 
environment  and  provides  a  shared  semantic  lexicon  for  communication. 
Robots  can  use  the  workspace  to  communicate  about  the  task  and 
environment  both  graphically  (e.g.  “The  highlighted  area  has  been 
searched!”)  and  verbally  (“Landmine  found  near  Victim  2!”)  using  the 
semantic  names  which  have  been  assigned  within  the  shared  cognitive 
workspace.  Conversely,  the  human  can  task  the  robots  in  much  the  same 
way. 


6.  CONCLUSION 

The  work  reported  on  here  represents  some  first  steps  towards  a  vision  of 
true  team  work  between  humans  and  robots.  We  believe  that  we  have 
developed  unique  answers  to  fundamental  problems  of  communication  and 
control,  which  will  permit  more  robust  team  interaction  between  humans  and 
dynamic  autonomy  robotic  forces.  The  hope  is  that  this  work  can  provide  the 
foundation  for  investigating  these  problems  further.  Already,  the  workspace 
is  invaluable  for  user  as  they  investigate  new  terrain,  especially  in  cluttered, 
labyrinthine  environments.  Future  experiments  with  human  subjects  will 
analyze  the  utility  of  the  collaborative  workspace  more  rigorously.  At  the 
current  time,  the  INEEL  is  working  to  add  additional  autonomous  tasks  such 
as  patrol  and  perimeter  surveillance  to  the  capabilities  of  the  control 
architecture.  Also,  much  work  is  currently  focused  on  task  specific 
perception  such  as  the  ability  to  more  accurately  recognize  Urban  Search  and 
Rescue  Victims.  As  these  capabilities  grow,  the  utility  of  the  collaborative 
workspace  grows  alongside. 
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